Freedom 4: The freedom to use the program effectively, efficiently and satisfactorily. A guide by Jan-Christoph Borchardt.
For a software to truly be free, people need to be able to easily use it without help. This is a quick start to usability testing in a distributed and independent development environment: A collection of usability methods applicable especially in free & open source software projects. No matter if you are a developer, usability person or just enthusiastic about making software easier to use, this document should help you get going.
This started out in Summer 2011 as a bachelor’s thesis concluding my studies of information design at Stuttgart Media University. The original title was Coordinating low-cost usability testing in independent free & open source software projects. Now it’s an open document, everyone can use it & contribute.
This guidebook is a result of being heavily involved with both free software and usability. Most of the learnings come from working on two free & open source software projects.
In Fall–Winter 2010 I interned at relevantive in Berlin and did user tests in the lab, in the car and by foot. I also got involved with OpenUsability.
I did interaction design for Cardapio, a menu and launcher similar to the Windows start menu and Mac OS Spotlight.
In Summer 2009, I worked together with Canonical’s Charline Poirier on usability testing for Shotwell before it became Ubuntu’s default photo application. The results are published on my blog.
This document aims to be concise and at the same time include the essential info which I find important in working on the projects. Rather than theory and analysis of the current state, which is already widely discussed, I want to compile a set of actionable methods because this is what I see as lacking at the moment.
It is in no way complete or fixed in its form, I encourage feedback and will evolve the document.
As references, I use openly available sources so everyone is able to read up more on the same documents I use. Also to honor authors who make their material available to the community, much like free software developers.
The subtitle is a reference to the four freedoms
of free software (Stallman, 1986).
This document is licensed under Creative Commons Attribution-ShareAlike:
You are free to share and remix the work as long as you attribute Jan-Christoph Borchardt as the author and distribute the resulting work under a similar license.
Most free software projects are developed by volunteers on their free time and don’t receive corporate backing. There is neither budget for dedicated usability tests nor particular usability expertise. Developers and designers decide from their point of view (Nichols & Twidale, 2003; Thomas, 2008).
Free software development is famously described as a bazaar
by Raymond (1997), a hierarchically flat organization. Especially independent projects are a meritocracy or do-ocracy where people are mainly valued according to their contributions to the project.
Open source development is a meritocracy: those who are judged as doing good (read: high quality) work wield power or influence over the community; the maintainers of the code give permission to check in software.
A problem with the organic and distributed bazaar workflow are the different communication channels, be it mailing lists, IRC (internet relay chat), bug reports, wiki pages and other platforms.
Discussion is fragmented between the mailing list and bug reports. These channels need a clearer charter and decision-making system.
There are many international contributors working on the software. This is not only code of the application itself but also translation, design, website, outreach and more. Already described by Benson, Müller-Prove & Mzourek (2004), we need a clearly defined decision-making process
to coordinate all the different efforts going in to designing and developing the product that is the software.
The integration of software cannot be achieved by committee, where everyone has to put in their own additions (featuritis again). It must be controlled by dictatorial artists with full say on the final cut.
And additions not only come from developers but also indirectly from users:
Reading dozens of GNOME and Red Hat bugs per day, I find that users ask for a preference by default.
At the same time, multiple interface designers need to agree on their designs. Decisions need to be documented and based on founded claims. It needs to be made clear that interface design is more of a science where research is important than an art that is subjective:
If various people work on the same interface, they will need to justify what they do, convince others why their ideas are so great and so on.
A guide on how to adapt to development structures, do user testing, use available free tools to ease recording and analysis as well as best practices for reporting and fixing usability issues.
Nichols & Twidale (2003) suggest conceptual methods like for example involving usability experts and students, creating a usability discussion infrastructure and evangelism. While I highly support that, I want to offer immediately actionable methods who can be carried out by everyone – much like a tutorial.
The focus is on formative and qualitative testing. Small tests with results based on actual behavior fit best with the fast and distributed development of free software projects. This enables to do quick iterations in design and development while receiving direct feedback on changes.
Classic usability testing, observing a person while they use the software, is of course one of the best ways to do assess usability – see 6.10 Watch people use the software. But especially in free software environments lots of other methods are more feasible.
The different methods are intended as patterns: Their usefulness depends on the project structure and the person responsible for the testing. They are guidelines and a collection of tips you can apply.
To work on the usability of a project you don’t need specific usability expertise – just the passion to learn and apply it. This is well described in the UX Advocate program started by David Siegel of Canonical and promoted by Allan Day of GNOME. Although it unfortunately didn’t gain enough traction, the motivations behind it are very true:
To be a UX Advocate, you don’t need to be able to create pixel-perfect mockups in Inkscape or have an HCI degree. All you need is love – you have to love an open source project and the people who use it, and you need to be patient, persistent, and persuasive. Of course, if you have some background in user experience, that would be tremendously helpful, but it’s unnecessary; it’s far better for an open source project to have a novice UX Advocate than none at all.
A UX Advocate doesn’t need to be a developer. They don’t even need to be a usability expert (though they can be on their way to becoming one). What they do need is the time, energy, and inclination to champion user experience. A UX Advocate can triage and prioritize UX bugs, and they can research design problems. They can even do user testing.
This document is exactly for making that as easy as possible.
If you’re a usability person looking for a project to improve, don’t pick any project but one useful to you. Independent free & open source software is about scratching one’s own itch (Raymond, 1997) so you should do it in a similar fashion as a user representative (Nichols & Twidale, 2003). You should be committed to the project and know it inside out.
The other developers are volunteers as much as you. Just jump in and do your best, your work will be appreciated. In contrast and as an advantage to paid usability work there is no constraint set by clients as to how the test should go. You can freely assess which methods you use and how many (or rather few) participants you interview.
Many free software developers already know about usability and the importance of it but don’t know how to improve it. A big issue in working with them is a lack of trust (Andreasen et al., 2006, p. 309). That is why you have to openly communicate your findings and methods. Ideally educate the developers about testing methods and common usability problems so they are enthusiastic about it themselves.
The earlier you do testing, the better. No developer likes throwing away lots of his code, especially those working in their free time. And this shouldn’t be the issue holding the software back from being more usable. Communicate your findings early and often (Reitmayr, Balazs & Mühlig, 2006).
Awesome that you read this! This document should give you ideas and provide you with methods on how to improve the usability of your software. Even if there is no designer on your project, you can use many of the research methods yourself.
One of the most important things in usability is that […] the product should target people whom [you] consider to be clueless newbies
(Trudelle, 2002). Everything should be as clear and self-evident as possible (Krug, 2008). You already know the interface inside out and probably even designed it. Hence you tend to either overlook or simply not notice issues (Johansson, 2005).
To learn more about general usability and develop an eye for issues, these blogs should go in your feed reader:
From searching information on the website and downloading the software over installation to actual usage, it is important to test every part.
Users are not only people directly using the software but also for example interested people browsing the website, contributors willing to help and administrators hosting it.
Especially in free & open source software, there are specific cases which appear more often than in proprietary software. For example setting it up on one’s own server when it is a web application. Or how to contribute to the project, an essence of free software. Also, support is mostly handled by the developers themselves rather than by a specific support team.
Translations are managed by volunteers and might be faulty. Use distributed user testing and encourage other people to test your software in other languages as well. See 6.11 Call for testing.
When people first hear about your project, they will check out the website. Many projects make the mistake of designing the website like it is a portal for developers or experienced users. Rather it should have information for people who never heard about the software before. Include short and understandable bits of information about what your project is.
Free software projects often have a wiki or use the project management system as their website. That is exactly the »portal for developers« issue. Even (or rather especially) a very bare page with only some info, screenshots, download and help links is better than a platform where all the information fights for exposure and the really important things are buried.
When the main website of a web application also acts a provider for the service, it needs to offer the relevant actions like Register and Log in as well. Especially these forms need to be made as easy as possible and tested thoroughly because they need to be »defeated« even before using the application itself.
With a website you ideally do a user test, see 6.10 Watch people use the software. You can let them go through the whole process:
This is of course the main focus of your work.
Regardless if your software is a web application or there are desktop applications available for multiple platforms such as free software distributions, Mac OS and Windows – you should test all of these if possible. With users of that specific operating system as participants you can see how people familiar with different interaction concepts handle your application.
More and more people today have smartphones and need the software usable on that – be it for Android, iOS, webOS, MeeGo, Windows Phone 7 or BlackBerry. Adjusting the web interface is the easiest route as there likely are no capacities for development of dedicated applications. If you use open protocols you might be able to recommend 3rd party software (we do that for Nextcloud and mobile WebDAV clients). In that case you need to test how it works and if there are multiple clients only recommend one – the best one.
Testing of basic functionality can be done beforehand. For example a web interface can be viewed in different browsers on the cross-browser testing site Browserling.
As important as the usability of the application itself is, always note that when the website and installation is not easy enough, people will never even come that far.
How do advanced users and potential administrators of your software handle setting up your software? How do regular users go through the installation process?
For web applications, the people setting the software up on their servers are most likely a completely different target group from the users themselves. Nevertheless technical terms should be reduced as well as options which are not immediately useful or can be set by default.
It is possible in free software operating systems to install a package directly from repositories onto the computer. Many tutorials describe step by step how to add a personal package archive to repositories or how to pull applications from a Git version control repository and compile them. Even non-technical people can easily get your software through these guides and you have to cater your installation process to them as well.
No matter how good you do your work, people will have problems, ask questions and report bugs. For them to be able to easily do that and also have the message reach you or the developers, there needs to be an easy platform.
Many free software projects use Bugzilla for issue tracking – it has a very difficult interface and a cumbersome registration process for reporting issues (Nichols & Twidale, 2003). A much better free software tool for support is Shapado which enables people to ask questions without registering.
Ideally, a help platform like that should be directly included in your software.
Helping hands are always important for a free software project. New contributors are needed so it should be easy to jump in. Be it writing code, designing, translating, writing docs, blogging or donations. Lowering the barrier to participation is essential.
Ideally there should be some »junior jobs« or quick opportunities for new people to get started in the code (Çetin et al., 2006). For that you need to mark bugs or tasks accordingly, see 7.4 Identify paper cuts.
The different communication and documentation channels need to be apparent and easily available. For programmers, the repository is important, for translators the language files or online translation tool, for others the IRC channel or mailing list.
The classic mailing list or IRC channel is not very approachable for people not already familiar with it. To still have that communication channel every seasoned developer is used to but also offer newcomers easy interfaces, there are two great free & open source web applications:
OnlineGroups offers mailing lists with a web interface. For people not familiar with mailing lists this is easily usable by the web interface alone.
freenode web IRC offers a web interface for IRC. People don’t even need a specific client, you can also just embed it on the website.
Always have pen and paper and note down any issues you discover. This can be when you use the software yourself (6.1 Dogfood), in user tests (6.10 Watch people use the software) or when talking to people (6.5 Benchmark opinions). To be able to assess the severity of issues later you can also note down how frequent and persistent they are (Nielsen, 2005b). See 7. Reporting.
There isn’t really much to prepare other than:
As mentioned in 3.2 Application itself already, test on different devices. Especially when the project is a web application, you need to make sure it works fine on big and small screens as well as touchscreens of tablets and smartphones.
When you don’t have access to a specific device, it’s very easy nowadays to find people who use them regularly.
Letting people use their own hardware has the upside that they won’t be distracted by any issues with the hardware or the software setup itself. Getting accustomed to your device will waste some time. You always have to consider that when issues come up.
When you use your own hardware always have clean installations of programs that need to be used, for example deactivate all browser plugins when you test a web application.
When you test a specific application make sure you have the latest version or development master. Having some knowledge of the version control system used by the project, for example Git, is also helpful. You can always pull the latest development version, check out new features and analyze their performance while they are developed.
In the case of a web application, you should host an instance of the software or organize one to be hosted. Prepare user accounts in case people are not comfortable registering.
For testing a desktop application on your own hardware, it is best if you set up a fresh installation. Then you can tell the participants that they can’t break anything and they will already be more relaxed even though it is unfamiliar hardware.
When testing on the participants’ hardware, you can go through the whole process of downloading, installation, first impression and identify more issues which naturally occur there.
Recording usability tests or interviews to have user quotes for presentations is standard practice in usability agencies. The participants need to sign release forms to agree to the videos being used.
In independent free software projects recording participants is rather useless because it creates much work just for presentation. Since we concentrate on small test sets with quick iterations this creates too much overhead. There should be no personal data or images and just recording of issues.
Also, reviewing the recordings afterwards uses up twice the time. It can be useful for longer tests though so here are some tools:
Define the target group of your project and get representative users to use the software. We want to test how usable the software is for normal people. As Benson, Müller-Prove & Mzourek (2004) advise: users on mailing lists [and IRC] are not typical users
but rather experienced ones.
It is enough to test with 3–5 users to find out most usability issues (Nielsen, 2000):
Nielsen also notes: The most striking truth of the curve is that zero users give zero insights.
So go get some participants and start testing!
Family members are easily reachable but most often not the best choice for testing. Either they know what you do and are thus technically inclined or they feel obliged to help you, making it a non-ideal situation for testing.
Friends, coworkers and flatmates are very good test participants as long as they are no designers. Friends and flatmates likely have lots of different professions and backgrounds, giving you various insights.
The problem with this group of people is that you can not conduct tests with the same people too often. Usually, interview participants are not invited again for the next three months and only three times in total. Otherwise they will become »professional usability testers« who just say what you want to hear and don’t act naturally.
Testing while traveling is great because you meet lots of different people. There is a vast array of potential test participants readily available. It doesn’t matter if it’s in a car, on a plane, bus or train, in a hostel or anywhere else you are staying or traveling.
You just have to strike up a conversation with someone. Ask them if they are interested in testing a software. For these unplanned tests you should have your own laptop with you. When they agree, just continue with 6.10 Watch people use the software.
When you don’t have that much time or the hardware for an interface test, a casual talk already works fine. The best method here is 6.5 Benchmark opinions where you just talk to people about the software they currently use and their experience with it.
The problem of people becoming »professional testers« will practically not occur since you mostly see the people you meet while traveling only once. There should be no problem to find 3–5 people with wildly varying professions and technological backgrounds.
Just chatting up random people at the mall or on the street (while sitting alone on a bench for example) might be off-putting at first but it will most likely be rewarded. Mozilla’s Jennifer Lynn Morrow made a great experience with user testing in the wild:
The mall is a fantastic place to find user test participants, because the range of technical expertise varies widely. Also, the people I encountered tended to be bored out of their minds, impatiently waiting for their partners to shop or friends to meet them.
She writes about her encounter and quick user test with a man who never used a computer before:
One of the hardest things to relate to Joe is the idea that you must first click in a text field in order to type.
This kind of person would rarely, if ever, come up in a standard usability test where people are recruited based on specific criteria. But they are incredibly insightful because they are not that biased based on previously used interfaces.
Research is not only classic usability testing. There are lots of other methods very useful and quick to assess the usability of your application.
It doesn’t really matter how much you know about usability – after working on a project for a couple of weeks you’ve become blind to many of the problems. Sure, experience will help you avoid making many design mistakes in the first place, but something will slip through. Every time. That’s why doing even a very basic usability test will improve your site.
You can not do enough user testing. Even when someone else already tested an aspect of your software, you have access to completely different people who might encounter completely different problems. It is very important to also test the improvements resulting from a previous usability test and iterate this development cycle.
Be genuinely interested in people, in the end your software is for them to use. Don’t make it feel like a test or an interview, much more a casual talk about their likes and dislikes. At the same time remember that you are looking for issues, not design advice or new features. It’s you who analyzes the problems and designs or gives suggestions to the developers. If people give design advice or feature suggestions, find out the underlying issue.
»Dogfooding« is a term in software development which refers to using the software you work on. And that is exactly what you should do as one of the easiest methods to find out usability issues: Use the software yourself as much as you can. Rely on it for what it does.
If you use the software in your daily workflow you will eventually experience more and more usability issues. You need to always cross-test them with actual non-technical users. While you might be a usability expert, the true experts of the interface are still the people using it.
The other developers should use the software regularly as well. Encourage everyone to use it and pay attention to issues they discover. When the developers themselves see the problems and stumble over them regularly, they are more likely to just fix them.
Similar to this method is 6.4 Benchmark other software which basically is dogfooding the other way around: Using a similar product to find out the advantages and disadvantages.
When you work on a software long enough, you tend to become blind for its errors and interaction issues (Johansson, 2005). You need to practice putting yourself in the mindset of an unexperienced user who just discovered your software.
Repeatedly set up a new installation or register a new account and go through the setup and first steps through the eyes of someone who never used the software before.
Prepare a scenario: It is useful to have a series of tasks so that the program is used as it normally would. This does not need to be too elaborate, just a few logical steps that cover what you want to test. For the first test of Shotwell, I went with:
You can always refine it for the next tests, the important thing is to get started! It is then best to follow up with actual user tests using the refined tasks to see how actual new users handle it. See 6.10 Watch people use the software.
Identify common interface and interaction mistakes by testing the software against a few simple interface heuristics.
The following set of 8 heuristics is derived from Nielsen (1994, 2005a) and Faaborg (2010a):
The same system can then be used in the project’s bug tracking system to group the errors. See 7.3 Tag bugs.
Look at comparable software that is in broad use, for example proprietary alternatives to your project. Use them and find out what they do better than your project. This practice is very similar to 6.1 Dogfood, where you intensively use your own software – but the other way around: You need to know why other software is so compelling and easy to use.
This can also help when you don’t know how to design a specific interaction. A project similar to yours might have already solved it elegantly. At the same time don’t imitate their design and eventually their mistakes (Thomas, 2008). See 6.8 Make things differently.
Also get the developers of your project to use the other software so they can experience it themselves. Developers need to know which alternatives there are and what leads people to prefer one over the other. Similar to dogfooding again, when developers see themselves what another software does good, they are more motivated to improve their own.
Talk to people what they think and especially don’t like about comparable software that is in broad use.
Most important is what they love and what they hate about their current software. What does in fact annoy them, what is completely awesome? Feature wishes are less interesting because they are future predictions you need to approach with caution (Nielsen, 2001).
This is possible nearly everywhere and with nearly everyone: Chat people up at lunch, riding the train, when using rideshare. Most like to talk about what annoys them if you have a genuine interest.
Subscribe to all online updates regarding your software. That way you can monitor problems, successes, what people love, what they hate and act upon them. Fixing it directly or helping with their problems is a good user experience by itself. There should be an official account for the software on the relevant platforms like Identi.ca, Diaspora and Twitter to be able to help or ask questions for clarification.
People will rant, praise, ask questions, post usage instructions for others or write a blog post describing how the software works. This is very insightful as outside perspectives on your project and how others perceive it to be.
I had the best results with the three searches of Identi.ca, Twitter and Google Blog. While you can view them as pages in the browser or search manually, you should subscribe to them in a news reader. Nextcloud News is a good free software, web-based feed reader I use for tracking Nextcloud news. These are the addresses for the different search sites where you need to replace »Nextcloud« with the name of your project:
At first it might seem like there is the problem of advanced users again. Are people who can use Identi.ca, Twitter and blogs really our target audience? Similar to what Benson, Müller-Prove & Mzourek (2004) say about mailing list participants. This is less of an issue here since the listed tools are much more common and easier to use than mailing lists.
When a problem or question occurs frequently, it needs to be documented and acted upon because it can be considered a severe usability issue.
Getting feedback can be as simple as that – just asking people what they love or hate. Really use the terms »love« and »hate« to get out the most severe things first. But you always have to filter out the actual issues from predictions as Nielsen (2001) notes. This is especially important with loud users
:
User feedback can be great, but some are more loud than representative and paint their needs and assessments as those of a supposed majority.
When regarding interfaces, always ask yourself: Are these elements important? How is the information used? Can we present it in a more useful way?
With an independent project, you have the chance to try out different interaction concepts and interface designs than in widespread or proprietary projects. When not many people use the software yet, you can experiment with designs and test if they are more usable than the current standard patterns. The latter might only be considered usable because they are familiar (Raskin, 1994).
A good example is the Chromium browser, which innovated against its already very widespread competitor Firefox. The Chromium developers removed the menu bar and replaced it with one wrench icon which reveals the menu. Mozilla’s Faaborg (2010b) then did research on menu item usage, following Firefox having a similarly condensed menu.
Strive to have a better design than other products, especially proprietary alternatives. Don’t just imitate their designs and replicate their mistakes (Nichols & Twidale, 2003; Thomas, 2008).
The earlier you find issues with your design, the easier it is to fix them. Every problem you fix with a pencil on paper is a problem you don’t have to fix in code.
To test the feasibility of a design in the planning phase, test mockups on people. Give them a realistic task, be general and don’t hint at the functionality of the software.
It is important to not only show the mockup to people but actually go through a scenario. Nielsen (2001) advises: If the users have not actually tried to use the designs, they’ll base their comments on surface features. Such input often contrasts strongly with feedback based on real use.
During and after the tasks you can ask them follow-up questions for them to clarify their impression of the design.
This is the classic usability testing method:
To discover which designs work best, watch users as they attempt to perform tasks with the user interface. This method is so simple that many people overlook it, assuming that there must be something more to usability testing. […] [U]ltimately, the way to get user data boils down to the basic rules of usability:
- Watch what people actually do.
- Do not believe what people say they do.
- Definitely don’t believe what people predict they may do in the future.
As Nielsen already describes, this is actually a very simple method – provided you follow some basic guidelines:
They can stop the test any time they want and you should stop it when you feel there are no more useful results. Time can vary between 10 and 90 minutes.
For each test you should use the same set of defined tasks to ensure that tests with multiple people are comparable. If you did 6.2 Be a new user before you can use the same tasks to also test against your personal perception of issues. This will at the same time help you to think more like a new user.
It’s best to start with a simple task to make participants feel at ease and act naturally. Make sure that the tasks are logical and follow a natural order.
When most of the participants run into a problem it can be considered major and needs to be fixed. For clarification of issues you should also ask the participants some questions after finishing the tasks:
For a demonstration of a quick usability test, see Steve Krug’s (2008) presentation on the least you can do about usability from 43:30 on.
Involving other people in testing helps in getting a wider array of user insights but at the same time needs to be coordinated correctly. Set up a page describing what you want to test, how to set everything up and introduce people to 6.10 Watch people use the software.
Ideally you should form a user experience team comprised of people interested in research and design. There you can have the same tasks for all interviews and compile the issues easier.
I urge you all to show Diaspora to friends and family without explaining any functions and watch them while they use it. Make note of any problems they have while interacting with the site. Watch what they do, disregard what they say (no design advice). You just did an usability test.
Canonical also did some distributed user testing for Ubuntu. A central person or team responsible for analyzing the results is needed to compile recommendations based on severity. They provide instructions on how to set up the test, how to conduct it and how to report the results. More information on their wiki.
For general feedback and to uncover more issues with new designs on the fly, there need to be people using the software regularly. Give the software to interested people, friends and family and encourage them to complain when there’s something annoying.
If the project is a web application, be sure to set up a server to be able to give out accounts. There you can always update with the latest versions and interface changes. In giving out the account, mention that it’s only a personal installation which you run for yourself and a few other people. As much as you take care, you can not guarantee against data loss so they should also store it elsewhere. Also say they should let you know when there are any problems.
Feedback might be hard to get, even when encouraging people to contact you about problems. Thus, when meeting someone who has been using the software, you should ask them about it or do a quick usability test and check how the current state of functionality and interface fares.
Longtime users uncover problems which are not immediately obvious in a usability test where the person uses the software for the first time only. Also, application developers dogfooding are so used to the basic functionality they might not even think about it anymore (Johansson, 2005).
I was actually very hesitant to include this since I highly value privacy. I kept it since results are anonymized and aggregated.
For the web application and web site of the project you can use the free & open source ClickHeat suite. It can be hosted on the same server and included in the site you want to monitor via JavaScript snippet.
This can help prioritize feature development and fixes. In a heatmap test with the public Nextcloud demo it was apparent that the search box was not used even a single time and is thus lower in priority – or not properly findable.
An advantage of free software development is the agile communication. You are able to report your results fast, without needing to compile a report or presentation. You can directly speak to the developers about it and make fast iterations based on suggestions. This allows for lots of small tests helping to gradually improve the interface.
Ideally, you should directly give suggestions on how to improve the interface. As Lewis (2006) mentions, usability issues need to be interpreted with caution. There should always be follow-up tests to check if the changes had a good effect on usability.
[…] usability practitioners must conduct their usability tests as carefully as possible, document their methods completely, and show proper caution when interpreting their results.
To prioritize issues, Nielsen (2005b) suggests rating their severity, judging from:
Openly communicate your findings with a list of usability issues.
Sort any issues according to the part of the application they occur in. This is important to find out which issues are present for multiple participants and should thus be considered major.
Categories could be:
It is not important who had a specific issue but much more how many people ran into it. If there is issue overlap, it is directly visible and can be marked as affecting more people.
When collaborating with more testers or in a user experience team, it is useful to use a collaborative editor like the free software web application EtherPad to collect results. You can get free team spaces and pads at the community-maintained TitanPad.
The next step from listing issues is analyzing the problems, derive suggestions from them and have actionable tasks to fix the issues.
The tasks should also be sorted by the section of the application they refer to. If the project has an issue tracking system where feature requests are tracked as well, use that for listing and prioritizing the tasks. If there are easy tasks, tag them accordingly. This will also attract potential contributors (Çetin et al., 2006).
Use tags in the project’s bug tracking system to identify and group issues according to common interface problems. This is similar to applying heuristics in review and you can use the same terminology for that. See 6.3 Review.
For this to work correctly not everyone should be able to set general »usability« tags but there needs to be a person or team coordinating the review and tagging of those bugs or supposed usability issues will clutter the tracker (Trudelle, 2002).
Existing bugs should be reviewed and identified according to their specific usability problem. If it is apparent that the fix for a problem is very trivial, additionally use the paper cut tag. See 7.4 Identify paper cuts.
Tagging bugs helps quantify usability issues and make them apparent to developers in an objective way:
Just as developers currently have a shared vocabulary to describe good and bad with concepts such as performance, data loss, and crashing, usability heuristics can introduce additional concepts, like consistency, jargon, and feedback.
Publicly dealing with usability in a way that is highly visible to developers contributes to the project being perceived as caring for user experience and can attract new developers interested in that.
The concept of paper cut usability bugs was developed by David Siegel, then on the user experience team at Canonical:
[…] a paper cut is a bug that will improve user experience if fixed, is small enough for users to become habituated to it, and is trivial to fix.
In accordance to what Johansson (2005) notes, it is mostly these small issues which are ignored by developers and experienced users but a pain for people who never used the software before.
To make it directly visible to contributors which issues are easily fixable, denote paper cuts when listing tasks or tagging in the bug tracker. While seasoned developers might care more about developing new things this is a good help for new contributors.
This initiative works very well for Ubuntu, they iterated it for every release as well as specific ones for important software packages. There is a dedicated Papercuts Ninja team for people interested in fixing paper cuts which acts as a starting point for new developers. See also the wiki page.
Similar to 7.3 Tag bugs, this outspoken focus on small usability bugs also helps to create general consideration for these issues.
Generalize the look and elements used in the application. It should be easy for existing developers and new contributors to make the interface consistent just by following some basic style guide.
You can find more sophisticated human interface guidelines for some projects available online:
While these are very extensive, in the beginning it is enough to document design decisions and their reasons to compile a quick design introduction.
When the user tests or personal experience reveal deeper problems with the design, sketching the suggestions is best to visualize them to other people on the projects. Even very quick and dirty mockups can help a lot.
This sketch started off the Nextcloud file manager redesign. I openly communicated it to the mailing list together with a writeup of changes and reasons.
While your design is not yet tested in implementation, be wary of people evaluating it from the mockup. With a fundamental design change many people will criticize it for being different than the current state (Nichols & Twidale, 2003). At the same time, consider that you didn’t see your designs in production yet either. You can only provide the reasoning which lead you to make the decisions and changing the design as you did. Also see 6.9 Test mockups.
This is important to be respected by the developers and be in accordance with the meritocracy. If you can fix issues yourself you are also not reliant on the developers doing it for you and can iterate even quicker when there is a small show-stopper.
Some technical knowledge of the project can also help in understanding underlying technical issues which may come up in usability testing. Then you can easily judge if it is a problem with the interaction concept of the application or a bug.
I want to make clear that design and software development belong together. That one flows into the other. That no good designer in this realm can stay entirely ignorant of implementation issues, and that no competent programmer can stay out of design.
While Nelson’s (1990) dictatorial artists with full say on the final cut
are needed to coordinate the design decisions, in an independent free software project everyone is a volunteer and does not want to be bossed around. If you engage in fixing issues yourself and not only report the problems, developers are more likely to acknowledge what you do.
Find a developer willing to partner with you on a combined usability session and coding sprint. You can have quick iterations to implement interface ideas, fix problems or evaluate the results of the usability testing session.
In the process, you can learn about development and the developer will learn about your work. In making your work directly visible to other developers you build trust and make it more likely for them to respect your suggestions and eventually implement them (Andreasen et al., 2006).
Do this one-on-one working with every main developer at least once to get them accustomed to user-centered design. That way you also learn how each developer thinks about the methodology and how you can help them make interfaces easier to use from the outset.
Establish a workflow for going from a planned or suggested feature over implementation to user research and improvements.
New functionality needs to be tested thoroughly, especially if it addresses a previously found usability issue. This is best done in cooperation with the developers to already evaluate the features before implementation and prioritize the results of research.
With new features it is important to discern actual usability issues from problems people have (or think they have) just because the interface is different than it was before. For that, you should ideally test with people who never used the old interface and thus don’t compare the differences.
This pattern was developed in collaboration with Ilya Zhitomirskiy of Diaspora. Also see Thomas (2008): Establish a process for usability specialists to contribute to a project.
A hackfest or sprint is a get-together of project contributors which can be a few hours or a few days, ideally on a weekend. There, everyone can talk directly to each other, proposals and problems can be discussed in the group. Developers can ask interface and usability people directly how to ideally implement something in the frontend.
When developers are in the same room, they can discuss interaction design using whiteboards, paper prototypes, spoken words, and gestures. But on the Internet, these often aren’t available, making discussions much slower and prone to misunderstandings.
Hackfests are ideal to present results of usability testing or reviews you did. You can display your work which is not necessarily measurable in lines of code. This helps to build the trust which is most important in cooperating with free software developers:
Relevantive stated that when there was no face-to-face contact with the other person, building trust could be the most strenuous task of co-operating with OSS developers.
There is a wide array of low-cost usability methods available which also very much fits in the free software development environment. It does not necessarily need usability experts to carry these methods out, just solid determination to make the software better.
I hope this document helps to increase awareness about the importance of user research in the free & open source software community and at the same time offer the tools to put it into practice. I encourage everyone to use the methods and give feedback on them. As said in the introduction, this document will be released publicly on my website at jancborchardt.wordpress.com and then constantly evolve with things I and other usability practitioners learn.
We in the free & open source software community should always keep in mind that we have one common goal: Freedom for people. And that freedom offered by the software is only available to users if the software is properly usable.
Andreasen, M. S., Nielsen, H. V., Schrøder, S. O., Stage, J. (2006). Usability in Open Source Software Development: Opinions and Practice. Retrieved 30 June 2011.
Balazs, B. (2011). Designer vs. Developer? Retrieved on 30 June 2011.
Benson, C., Müller-Prove, M., Mzourek, J. (2004). Professional Usability in Open Source Projects: GNOME, OpenOffice.org, NetBeans. Retrieved 30 June 2011.
Çetin, G., Erkan, K., Verzulli, D., Ozirkovskyy, L. (2006). Usability Involvement in F/Oss Projects. Retrieved 30 June 2011.
Day, A. (2010). User Experience Advocates. Retrieved 30 June 2011.
Faaborg, A. (2010a). Quantifying Usability: How injecting usability principles into standard bug tracking software can reshape how your organization approaches UX design. Retrieved 30 June 2011.
Faaborg, A. (2010b). Firefox 3.5 & 3.6 Menu Item Usage Study. Retrieved 30 June 2011.
Frishberg, N., Dirks, A. M., Benson, C., Nickell, S., Smith, S. (2002). Getting to Know You: Open Source Development Meets Usability. Retrieved 30 June 2011.
Johansson, R. (2005). Usability testing without a budget. Retrieved 30 June 2011.
Krug, S. (2008). The least you can do about usability. Retrieved 30 June 2011.
Lewis, J. R. (2006). Usability Testing. Retrieved on 30 June 2011.
Mathis, L. (2011). Designed for Use. Retrieved on 24 August 2011.
Morrow, J. L. (2011). User Testing in the Wild: Joe’s First Computer Encounter. Retrieved 7 July 2011.
Nelson, T. (1990). The Right Way to Think About Software Design. In Laurel, B. (Ed.), The Art of Human-Computer Interface Design (p. 235-243). Reading: Addison-Wesley. Retrieved 30 June 2011.
Nichols, D. M., Twidale, M. B. (2003). The Usability of Open Source Software. Retrieved 30 June 2011.
Nielsen, J. (2005a). Ten Usability Heuristics. Retrieved on 30 June 2011.
Nielsen, J. (2005b). Severity Ratings for Usability Problems. Retrieved 30 June 2011.
Nielsen, J. (2001). First Rule of Usability? Don’t Listen to Users. Retrieved on 30 June 2011.
Nielsen, J. (2000). Why You Only Need to Test with 5 Users. Retrieved 30 June 2011.
Nielsen, J. (1994). Heuristic evaluation. In Nielsen, J., and Mack, R.L. (Eds.), Usability Inspection Methods. New York: John Wiley & Sons.
Pennington, H. (2002). Free software UI. Retrieved 30 June 2011.
Raskin, J. (1994). Intuitive Equals Familiar. Retrieved 30 June 2011.
Raymond, E. S. (1997). The Cathedral and the Bazaar. Retrieved on 30 June 2011.
Reitmayr, E., Balazs, B., Mühlig, J. (2006). Integrating Usability with Open Source Software Development: Case Studies from the Initiative OpenUsability. In özel, B., Çilingir, C. B., Erkan, K. (Eds.), Oss 2006 tOssad workshop proceedings (p. 65–72). Retrieved 30 June 2011.
Siegel, D. (2010). Announcing the User Experience Advocates Project. Retrieved 30 June.
Siegel, D. (2009). Paper Cut. Retrieved 30 June 2011.
Stallman, R. (1986). The Free Software Definition. Retrieved 30 June 2011.
Thomas, M. P. (2008). Why Free Software has poor usability, and how to improve it. Retrieved on 30 June 2011.
Trudelle, P. (2002). Shall We Dance? Ten Lessons Learned from Netscape’s Flirtation with Open Source UI Development. Retrieved 30 June 2011.
Wilms, T. (2011). Design-Boost. Retrieved on 30 June 2011.
To all the amazing people who helped me (even unknowingly) in writing this:
Nextcloud: Frank Karlitschek, Robin Appelman, Jakob Sack, Michael Gapczynski, Arthur Schiwon, all other contributors …
Diaspora: Ilya Zhitomirskiy, Kevin Kleinman, …
MediaGoblin: Chris Webber, Joar Wandborg, Deborah Nicholson, Jef van Schendel, …
Gnome designers: Allan Day, William Jon McCann, Hylke Bons, Jakub Steiner, Calum Benson, Andreas Nilsson, Máirín Duffy, …
Ubuntu designers: Charline Poirier, Matthew Paul Thomas, David Siegel, …
relevantive: Fabian Klenk, Dominik Sibum, Jan Mühlig, …
Media University Stuttgart: Michael Burmester, Ralph Tille, David Prüm, …
Michiel de Jong of Unhosted, Bjarni Rúnar Einarsson of PageKite, Thiago Teixeira of Cardapio, Samuel Clay of NewsBlur, the Shotwell team, Peter Sikking of GIMP, Ellen Reitmayr, Parker Higgins, Max Heinemeyer, …
Everyone who participates in a usability test and provides valuable feedback.
…